This package contains tools to help diagnose connectivity problems within
a Myrinet fabric.  There are three main programs, "watch_switches", 
"wirelist" and "link_test".

watch_switches samples Myrinet switch counters every several seconds,
and prints out differences in the counters, such as the badcrc counts.

wirelest is used to create a human readable wiring list of the fabric.  
link_test will examine every interswitch link in the system and report
back which ones are failed completely or are marginal.  The output from
wirelist is necessary to convert the output of link_test into connections
meaningful to humans.  A discussion of running each program is given 
below.

There are 2 other programs: poke and xecho. poke sends a message down
any route and xecho sends either an X32 ID packet or a raw packet 
down a route and then back again.

======== building ==========

Take linktest.tgz, untar it.  It will create a directory called "src".

If GM is not installed in /opt/gm, then edit "src/link_test/Makefile and
change the 2nd line to be "GM=<dir_where_GM_installed>".

Then:
$ cd ../libldiag
$ make
$ cd ../tools
$ make

This will create three executables of interest, "watch_switches", 
"wirelist" and "link_test", and xecho and poke.

======== running watch_switches ==========

Create a list of all the switches in <switchfile>, then run:
    watch_switches -s <switchfile>

You will get output about changing counters in the switches.  If there
is too much output, try increasing the threshold using "-T".  Full usage is:
    watch_switches -s <switchfile> [ -T <threshold> ]
	[ -n <sample_period> ] [ -d ] [ -g ] [ -D <tmpdir> ]

-T = only show counts which increased by > <threshold> during the
	last sample period (default 0)
-n = sample counters every <sample_period> seconds (default 5)
-d = give a report of all manually disabled links
-g = show goodCRC counts (off by default since counts are typically large)
-D = a temporary working dir used to store switch output (default /tmp/watchdir)


======== running wirelist ==========

wirelist takes a map file and a list of monitoring line card addresses
and creates a human-readable wiring list of the fabric.  You should have
no mappers running anywhere on the fabric when you run this.

wirelist MUST be run on a node which is both connected to the Myrinet
fabric and has IP connecticity to the monitoring line cards on the switches.

The map file is one you get from running the mapper and specifying that a 
map file be created via --map-file-0=<map_file>

To run wirelise:
$ ./wirelist -m <map_file> -s <switch_list> -W <wire_list>

<switch_list> is a file containing the names or IP addresses of each monitoring
line card in the fabric, one per line.

<wire_list> is the output wiring list file.

wirelist may need to be run once or twice before it completes successfully.
If you have troubles with it, please contact reese at reese@myri.com.


======== running link_test ==========

link_test tests every switch-switch link in the network.  It does not
test the host-switch links as the NIC firmware is alread pretty good at
reporing problems there.

To run it, you need the map file for the cluster. It does not need to have all
the hosts in it, but it does need to have the host from which link_test
is being run.

First, to verify connectivyty, run:
$ ./link_test -m <map_file> -l 4


Then run a couple of times:
$ ./link_test -m <map_file> -l 8000

The first one tests the links with 4k of data, quick and useful for finding
completely dead links.  The second tests with 8000k of data over each link,
and is useful for finding links with medium-high packet loss.  The output
should tell us exactly which links are bad.

The number 8000 can be increased to look for links that have lower error rates
which may not show up with only a few thousand packets.

The output of link_test will specify problem links with output that looks
something like:
  x34:4 <-> x17:2

This refers to the link between xbar 34, port 4 and xbar 17, port 2.  This
can be converted into something useful to a human by crossreferencing the
xbar:port with the wiring list generated by wirelist.  In this example, search
for "x34:4" in the wirelist output to see which link is having a problem.

You can also test a link on a crossbar32 enclosure using X32 Identification
packets. This is done with the -I option. Note that link_tst -I option will
fail in all of the crossbars in the fabric are not xbare32's. 

====== SW32 (xbar32) compatability =====

link_test supports enclosures with xbar32 switches (M3-SW32-16F, M3-2SW32,
M3-4SW32-16Q). Use the "-I <num pkts>" option on pure xbar32 clusters using
XBAR32. This will only work if all port line cards are of the *SW32* variety.
If upper quadrant routing is enabled on  all of the xbar32's (_not_ the
default) you can also use link_test in the normal mode, with the "-l"
switch. If all of the xbar32's have upper quadrant routing enabled,
link_test with the -l switch will work in mixed xbar16, xbar32
environments. In order to disable upper quadrant routing, you have
to use the web interface on the monitor card.

wirelist and watch_switches currently do not run on xbar32 equipment.

========= example ==============

The following illustrates use of link_test to find a dead link:

/* -l 4 sends just a few packets to verify link connectivity */
/* This run reveals one dead link */

# ./link_test -m map -l 4
Successfully opened board 0, mapper port
All OK! 1285 hosts, 560 xbars
I am host h1273 (1273)
b8 86 88 80 b8 ba 88
Received 0 of 808 packets (dropped 0?)
BW = 0.00MB/s
time since last rx = 20514 usec
x118:0 <-> x20:8 failed test.

3836 links checked, 0 retries.


/* Now, find the bad link in the wirelist */
/* This reveals the link between 10.0.0.7, slot 1, port 0 and */
/* 10.0.0.30 slot 7 port 8 is the problem link. */

# grep -C 8 x118:0 wire.out

Enclosure 10.0.0.6 Slot 38 serial n/a model UNKNOWN x147

Enclosure 10.0.0.6 Slot 39 serial n/a model UNKNOWN x179

Enclosure 10.0.0.6 Slot 40 serial n/a model UNKNOWN x537

Enclosure 10.0.0.7 Slot 1 serial 157825 model M3-SPINE-8F
Port 0 (x118:0) switch 10.0.0.30 slot 7 port 8
Port 1 (x150:0) switch 10.0.0.30 slot 11 port 9
Port 2 (x182:0) switch 10.0.0.30 slot 15 port 10
Port 3 (x91:0) switch 10.0.0.30 slot 3 port 12
Port 4 (x123:0) switch 10.0.0.30 slot 7 port 13
Port 5 (x155:0) switch 10.0.0.30 slot 11 port 14
Port 6 (x187:0) switch 10.0.0.30 slot 15 port 15
Port 7 (x547:0) switch 10.0.0.31 slot 3 port 9


